<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Using the Facemark API</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d3/d81/tutorial_contrib_root.html">Tutorials for contrib modules</a></li><li class="navelem"><a class="el" href="../../d5/d47/tutorial_table_of_content_facemark.html">Tutorial on Facial Landmark Detector API</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Using the Facemark API </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h2>Goals </h2>
<p>In this tutorial will helps you to</p>
<ul>
<li>Create a Facemark object.</li>
<li>Set a user defined face detector for the facemark algorithm</li>
<li>Train the algorithm.</li>
<li>Use the trained model to detect the facial landmarks from a given image.</li>
</ul>
<h2>Preparation </h2>
<p>Before you continue with this tutorial, you should download the dataset of facial landmarks detection. We suggest you to download the helen dataset which can be retrieved at <a href="http://www.ifp.illinois.edu/~vuongle2/helen/">http://www.ifp.illinois.edu/~vuongle2/helen/</a> (Caution! The algorithm requires around 9GB of RAM to train on this dataset).</p>
<p>Make sure that the annotation format is supported by the API, the contents in annotation file should look like the following snippet: </p><div class="fragment"><div class="line">version: 1</div><div class="line">n_points:  68</div><div class="line">{</div><div class="line">212.716603 499.771793</div><div class="line">230.232816 566.290071</div><div class="line">...</div><div class="line">}</div></div><!-- fragment --><p>The next thing to do is to make 2 text files containing the list of image files and annotation files respectively. Make sure that the order or image and annotation in both files are matched. Furthermore, it is advised to use absolute path instead of relative path. Example to make the file list in Linux machine </p><div class="fragment"><div class="line">ls $PWD/trainset/*.jpg &gt; images_train.txt</div><div class="line">ls $PWD/trainset/*.pts &gt; annotation_train.txt</div></div><!-- fragment --><p>example of content in the images_train.txt </p><div class="fragment"><div class="line">/home/user/helen/trainset/100032540_1.jpg</div><div class="line">/home/user/helen/trainset/100040721_1.jpg</div><div class="line">/home/user/helen/trainset/100040721_2.jpg</div><div class="line">/home/user/helen/trainset/1002681492_1.jpg</div></div><!-- fragment --><p>example of content in the annotation_train.txt </p><div class="fragment"><div class="line">/home/user/helen/trainset/100032540_1.pts</div><div class="line">/home/user/helen/trainset/100040721_1.pts</div><div class="line">/home/user/helen/trainset/100040721_2.pts</div><div class="line">/home/user/helen/trainset/1002681492_1.pts</div></div><!-- fragment --><h2>Creating the facemark object </h2>
<div class="fragment"><div class="line">/*create the facemark instance*/</div><div class="line">FacemarkLBF::Params params;</div><div class="line">params.model_filename = &quot;helen.model&quot;; // the trained model will be saved using this filename</div><div class="line">Ptr&lt;Facemark&gt; facemark = FacemarkLBF::create(params);</div></div><!-- fragment --><h2>Set a custom face detector function </h2>
<p>Firstly, you need to create your own face detector function, you might also need to create a <code>struct</code> to save the custom parameter. Alternatively, you can just make these parameter hard coded within the <code>myDetector</code> function. </p><div class="fragment"><div class="line">struct Conf {</div><div class="line">    cv::String model_path;</div><div class="line">    double scaleFactor;</div><div class="line">    Conf(cv::String s, double d){</div><div class="line">        model_path = s;</div><div class="line">        scaleFactor = d;</div><div class="line">        face_detector.load(model_path);</div><div class="line">    };</div><div class="line"></div><div class="line">    CascadeClassifier face_detector;</div><div class="line">};</div><div class="line">bool myDetector(InputArray image, OutputArray faces, Conf *conf){</div><div class="line">    Mat gray;</div><div class="line"></div><div class="line">    if (image.channels() &gt; 1)</div><div class="line">        cvtColor(image, gray, COLOR_BGR2GRAY);</div><div class="line">    else</div><div class="line">        gray = image.getMat().clone();</div><div class="line"></div><div class="line">    equalizeHist(gray, gray);</div><div class="line"></div><div class="line">    std::vector&lt;Rect&gt; faces_;</div><div class="line">    conf-&gt;face_cascade.detectMultiScale(gray, faces_, conf-&gt;scaleFactor, 2, CASCADE_SCALE_IMAGE, Size(30, 30) );</div><div class="line">    Mat(faces_).copyTo(faces);</div><div class="line">    return true;</div><div class="line">}</div></div><!-- fragment --><p>The following snippet demonstrates how to set the custom detector to the facemark object and use it to detect the faces. Keep in mind that some facemark object might use the face detector during the training process.</p>
<div class="fragment"><div class="line">Conf config(&quot;../data/lbpcascade_frontalface.xml&quot;, 1.4);</div><div class="line">facemark-&gt;setFaceDetector(myDetector, &amp;config); // we must guarantee proper lifetime of &quot;config&quot; object</div></div><!-- fragment --><p>Here is the snippet for detecting face using the user defined face detector function.</p>
<div class="fragment"><div class="line">Mat img = imread(&quot;../data/himym3.jpg&quot;);</div><div class="line">std::vector&lt;cv::Rect&gt; faces;</div><div class="line">facemark-&gt;getFaces(img, faces, config);</div><div class="line">for(int j=0;j&lt;faces.size();j++){</div><div class="line">    cv::rectangle(img, faces[j], cv::Scalar(255,0,255));</div><div class="line">}</div><div class="line">imshow(&quot;result&quot;, img);</div><div class="line">waitKey(0);</div></div><!-- fragment --><h2>Training a facemark object </h2>
<ul>
<li>First of all, you need to set the training parameters <div class="fragment"><div class="line">params.n_landmarks = 68; // number of landmark points</div><div class="line">params.initShape_n = 10; // number of multiplier for make data augmentation</div><div class="line">params.stages_n=5; // amount of refinement stages</div><div class="line">params.tree_n=6; // number of tree in the model for each landmark point</div><div class="line">params.tree_depth=5; //he depth of decision tree</div><div class="line">facemark = FacemarkLBF::create(params);</div></div><!-- fragment --></li>
<li>And then, you need to load the file list from the dataset that you have prepared. <div class="fragment"><div class="line">std::vector&lt;String&gt; images_train;</div><div class="line">std::vector&lt;String&gt; landmarks_train;</div><div class="line">loadDatasetList(&quot;images_train.txt&quot;,&quot;annotation_train.txt&quot;,images_train,landmarks_train);</div></div><!-- fragment --></li>
<li>The next step is to add training samples into the facemark object. <div class="fragment"><div class="line">Mat image;</div><div class="line">std::vector&lt;Point2f&gt; facial_points;</div><div class="line">for(size_t i=0;i&lt;images_train.size();i++){</div><div class="line">    image = imread(images_train[i].c_str());</div><div class="line">    loadFacePoints(landmarks_train[i],facial_points);</div><div class="line">    facemark-&gt;addTrainingSample(image, facial_points);</div><div class="line">}</div></div><!-- fragment --></li>
<li>execute the training process <div class="fragment"><div class="line">/*train the Algorithm*/</div><div class="line">facemark-&gt;training();</div></div><!-- fragment --></li>
</ul>
<h2>Use the trained model to detect the facial landmarks from a given image. </h2>
<ul>
<li>First of all, load the trained model. You can also download the pre-trained model in this link <a href="https://raw.githubusercontent.com/kurnianggoro/GSOC2017/master/data/lbfmodel.yaml">https://raw.githubusercontent.com/kurnianggoro/GSOC2017/master/data/lbfmodel.yaml</a> <div class="fragment"><div class="line">facemark-&gt;loadModel(params.model_filename);</div></div><!-- fragment --></li>
<li>Detect the faces <div class="fragment"><div class="line">facemark-&gt;getFaces(img, faces, config);</div></div><!-- fragment --></li>
<li>Perform the fitting process <div class="fragment"><div class="line">std::vector&lt;std::vector&lt;Point2f&gt; &gt; landmarks;</div><div class="line">facemark-&gt;fit(img, faces, landmarks);</div></div><!-- fragment --></li>
<li>Display the result <div class="fragment"><div class="line">for(int j=0;j&lt;faces.size();j++){</div><div class="line">    face::drawFacemarks(img, landmarks[j], Scalar(0,0,255));</div><div class="line">}</div><div class="line">imshow(&quot;result&quot;, img);</div><div class="line">waitKey(0);</div></div><!-- fragment --> </li>
</ul>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:38 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
